this is what my config.yaml looks like. I wasn't sure if we need to add:

... && versionGE(split(Target.CondorVersion)[1], "24.8.0") && (isenforcingdiskusage =!= true)'
stream_ouput: true
stream_error: true

parts to the documentation

# Default configuration for the SPRAS/HTCondor executor profile. Each of these values
# can also be passed via command line flags, e.g. `--jobs 30 --executor htcondor`.

# NOTE: File paths in here should be relative to where you submit from, typically the
# root of the SPRAS repository

# 'jobs' specifies the maximum number of HTCondor jobs that can be in the queue at once.
jobs: 30
executor: htcondor
configfile: config/egfr.yaml
htcondor-jobdir: htcondor/logs

# Indicate to the plugin that jobs running on various EPs do not share a filesystem with
# each other, or with the AP.
shared-fs-usage: none
# Distributed, heterogeneous computational environments are a wild place where strange things
# can happen. If something goes wrong, try again up to 2 times. After that, we assume there's
# a real error that requires user/admin intervention
retries: 2

# Default resources will apply to all workflow steps. If a single workflow step fails due
# to insufficient resources, it can be re-run with modified values. Snakemake will handle
# picking up where it left off, and won't re-run steps that have already completed.
default-resources:
  job_wrapper: "htcondor/spras.sh"
  # If running in CHTC, this only works with apptainer images
  # Note requirement for quotes around the image name
  container_image: "test-htc.sif"
  universe: "container"
  # The value for request_disk should be large enough to accommodate the runtime container
  # image, any additional PRM container images, and your input data.
  request_disk: "16GB"
  request_memory: "12GB"
  retry_request_memory_increase: "RequestMemory + 4"
  retry_request_memory_max: "32GB"
  classad_WantGlideIn: true
  requirements: |
    '(HAS_SINGULARITY == True) && (Poolname =!= "CHTC") && versionGE(split(Target.CondorVersion)[1], "24.8.0") && (isenforcingdiskusage =!= true)' 
  stream_ouput: true
  stream_error: true

I definitely don't think we should be covering the stream_{output/error} in the general purpose profile for fear that an unknowing user will take that as the default (and make a CHTC sys admin very sad when they crash an AP). These should be intentionally hard to fine because users will find them very tempting to use without understanding the detrimental effects they can have on shared computing resources.

As for the other requirements, I'd also like to avoid sticking those in the profile -- they're very specific to your run that needs both the OSPool and profiling, and these requirements are generally documented elsewhere.

What was the goal of these settings for Neha's OSPool runs? I don't think we need to add them here but am curious for our SPRAS benchmarking in OSPool.

• classad_WantGlideIn: true configures a classad that enables submission to OSPool
• (HAS_SINGULARITY == True) makes sure you land on OSPool EPs that support apptainer/singularity, which is a hard requirement for SPRAS
• (Poolname =!= "CHTC") -- this can probably be omitted -- it disables submission to CHTC EPs. If you enable OSPool submissions and don't include this, you're submitting to both the OSPool and CHTC at once.
• versionGE(split(Target.CondorVersion)[1], "24.8.0") && (isenforcingdiskusage =!= true) makes sure you match with EPs that have the features/configuration needed for the apptainer profiling code to work.

The next section is what I found confusing. It gives instructions to create the .sif from the existing DockerHub image. That usually breaks. I recommend we remove it and only give instructions to build a new image from source.

Maybe you can explain how these steps break for you? I don't think I've ever run into issues building from an existing Dockerhub image, outside of the mismatched Snakefile conundrum (which I try to cover more heavily in my latest round of revisions).

The new option to checkout a version of the SPRAS repo that matches the existing image makes sense and should help.

My prior confusion was around what a user should do if they need to build a SPRAS image themselves to match a newer commit. I don't see a way to build the Apptainer sif image entirely on CHTC. We link to CHTC docs that expect you to have a def file, which we don't have. The Apptainer instructions for building directly from a Dockerfile didn't work for me. My understanding is that I would have to build a Docker image on a local machine with Docker, push it to my DockerHub, then run a build job in CHTC to convert it to a sif file.